Analyzing Temporal Patterns in Behavioral Data with Liquid State Machines

We propose the use of brain inspired machine learning to complement statistical tests currently used in the field of behavioral marketing research. We analyze a data set of facial emotion responses from participants who watched a commercial. Our goal is to predict behavioral intentions of the participants from the temporal patterns of their facial emotions. We suggest interpreting the data as a regression problem with a temporal component, allowing machine learning algorithms to discover which facial emotions are important for behavioral intentions and when these features play a decisive role. We first set a baseline performance collapsing the temporal component of the regression analysis by taking the average facial emotions over the whole video or by taking the difference between the averages of the first and second half of the commercial. Thereafter, we apply a Liquid State Machine (LSM) to analyze temporal patterns. The LSM contains a recurrent neural network (RNN), which produces states with implicit temporal information and higher order combinations of features. The states of the RNN are regressed onto behavioral intentions with Support Vector Regression (SVR). We apply forward and backward feature selection to explain which facial emotions have most impact on predicting the behavioral intentions. Moreover, at several points in time we inspect the performance of the LSM to expose when important effects during the commercial take place. Optimizations in the notoriously difficult parameter search for the LSM are suggested. We investigate several different topologies to improve the consistency of the general performance of the RNN. Results suggest that “Happiness” is the most important facial emotion for the investigated dataset and that the optimal decision time to predict behavioral intentions is at 80% in the commercial. We compare our results to traditional statistics in behavioral marketing and show the added value of using machine learning to include analysis of temporal patterns. Finally, we suggest that our proposed methods can be applied in similar fields in which data with complex temporal patterns are found.